GoF/GoP texture description method, and texture-based GoF/GoP retrieval method and apparatus using the same

ABSTRACT

A method of describing texture of a group of frames (GoF) or a group of pictures (GoP) using homogeneous texture descriptors, and a method and apparatus for retrieving a GoF/GoP using the texture description method are provided. The texture description method includes: generating homogeneous texture descriptors of all frames constituting the GoF or all pictures constituting the GoF; and expressing the GoF or GoP using a predetermined representative homogeneous texture descriptor corresponding to one frame or picture to reduce the amount of data. The GoF/GoP retrieval method includes: establishing a database of homogeneous texture descriptors of a plurality of GoFs&#39;s or GoPs&#39;s, each GoF or GoP being expressed by a predetermined representative homogeneous texture descriptor corresponding to one frame or picture to reduce the amount of data; generating a homogeneous texture descriptor corresponding to one frame or picture of a query GoF or GoP when the query GoF or GoP is input; searching homogeneous texture descriptors that are similar to the homogeneous texture descriptor of the query GoF or GoP in the database; and retrieving GoFs&#39;s or GoPs&#39;s corresponding to the searched similar homogeneous texture descriptors and arranging GoFs&#39;s or GoPs&#39;s in the order of degree of similarity. Therefore, the texture of images can be more accurately expressed, and an image can be more efficiently and rapidly retrieved.

This application claims the benefit of U.S. Provisional PatentApplication No. 60/487,945, filed on Jul. 18, 2003, in the U.S. Patentand Trademark Office, and Korean Patent Application No. 2004-54719,filed on Jul. 14, 2004, in the Korean Intellectual Property Office, thedisclosures of which are incorporated herein in their entirety byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing, and moreparticularly, to a GoF/GoP (Group of Frames/Group of Pictures) texturedescription method and a texture-based GoF/GoP retrieval method andapparatus using the same.

2. Description of the Related Art

Texture information of images as an indicator of important visualfeatures of images has been studied for a long time. This textureinformation of images is used as a major low level descriptor forindexing and summarizing image or video data on a contents basis. Thetexture information of images is useful for retrieving a particularphotograph from electronic albums or for retrieving data based oncontents from tile or textile databases.

However, in order to retrieve a GoF corresponding to a video sequence ora GoP corresponding to a group of pictures in an images database, alldescriptors of individual frames constituting the GoF or individualpictures constituting the GoP have to be used as queries. For example,with the assumption that there is a GoF consisting of 100 frames, inorder to retrieve the GoF from an images database, the images databaseis searched using all descriptors of individual frames constituting theGoF as queries. Therefore, the data of the queries becomes too large,and it takes much time to retrieve data from the database using thequeries.

SUMMARY OF THE INVENTION

The present invention provides a method of describing texture of a groupof frames (GoF) or a group of pictures (GoP) using homogeneous texturedescriptors.

The present invention provides a method and apparatus for retrieving aGoF/GoP from an images database using the homogeneous texturedescriptor-based GoF/GoP texture description method.

According to an aspect of the present invention, there is provided amethod of describing texture of a GoF or GoP, the method comprising:generating homogeneous texture descriptors of all frames constitutingthe GoF or all pictures constituting the GoF; and expressing the GoF orGoP using a predetermined representative homogeneous texture descriptorcorresponding to one frame or picture to reduce the amount of data.

Components of each of the homogeneous texture descriptors may includethe mean and the variance of energies of frequency domains of aGarbor-filtered image, the energy of a DC channel, and the variance ofall pixel values of the image.

The predetermined representative homogeneous texture descriptor in theexpressing the GoF or GoP may include the average of the values of eachof the components of the homogeneous texture descriptors for all theframes or pictures, the median among the values of each of thecomponents of the homogeneous texture descriptors for all the frames orpictures, or the smallest among the values of each of the components ofthe homogeneous texture descriptors for all the frames or pictures.

According to another aspect of the present invention, there is provideda method of retrieving a GoF or GoP that is similar to a query GoF orGoP, the method comprising: establishing a database of homogeneoustexture descriptors of a plurality of GoFs's or GoPs's, each GoF or GoPbeing expressed by a predetermined representative homogeneous texturedescriptor corresponding to one frame or picture to reduce the amount ofdata; generating a homogeneous texture descriptor corresponding to oneframe or picture of a query GoF or GoP when the query GoF or GoP isinput; searching homogeneous texture descriptors that are similar to thehomogeneous texture descriptor of the query GoF or GoP in the database;and retrieving GoFs's or GoPs's corresponding to the searched similarhomogeneous texture descriptors and arranging GoFs's or GoPs's in theorder of degree of similarity.

The predetermined representative homogeneous texture descriptor in theestablishing the database may include the average of the values of eachof the components of the homogeneous texture descriptors for all theframes or pictures, the median among the values of each of thecomponents of the homogeneous texture descriptors for all the frames orpictures, or the smallest among the values of each of the components ofthe homogeneous texture descriptors for all the frames or pictures.

According to another aspect of the present invention, there is providedan apparatus for retrieving a GoF or GoP that is similar to a query GoFor GoP, the apparatus comprising: a homogeneous texture descriptordatabase storing homogeneous texture descriptors of a plurality ofGoFs's or GoPs's, each GoF or GoP being expressed by a predeterminedrepresentative homogeneous texture descriptor corresponding to one frameor picture to reduce the amount of data; a query homogeneous texturedescriptor generating unit generating a predetermined homogeneoustexture descriptor of a query GoF or GoP when the query GoF or GoP isinput; a homogeneous texture descriptor search unit searching thedatabase for homogeneous texture descriptors that are similar to thepredetermined homogeneous texture descriptor of the query GoF or GoP;and a GoF/GoP retrieving unit retrieving GoFs's or GoPs's correspondingto the searched similar homogeneous texture descriptors and arrangingthe GoFs's or GoPs's in the order of degree of similarity.

According to another aspect of the present invention, there is provideda computer readable medium having embodied thereon a computer programfor any one of the above-described methods.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of the present inventionwill become more apparent by describing in detail exemplary embodimentsthereof with reference to the attached drawings in which:

FIG. 1 is a flowchart illustrating a GoF/GoP texture description methodaccording to the present invention;

FIG. 2 is a view for explaining a process of expressing a GoF/GoP usinga predetermined representative homogeneous texture descriptor (HTD) toreduce the amount of data;

FIG. 3 is a block diagram of a GoF/GoP description apparatus using HTDsaccording to the present invention; and

FIG. 4 is a flowchart illustrating a GoF/GoP retrieval method using HTDsaccording to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

A GoF/GoP texture description method using homogeneous texturedescriptors (HTDs) and a GoF/GoP retrieval method and apparatus usingthe texture description method according to the present invention willbe described in detail with reference to the appended drawings.

Referring to FIG. 1, which is a flowchart of a GoF/GoP texturedescription method according to the present invention, when an arbitraryGoF or GoP is input, homogeneous texture descriptors (HTDs) of allframes constituting the GoF or of all pictures constituting the GoP aregenerated (operation 100). For example, if the input GoF consists of 100frames, 100 HTDs are generated for the GoF.

An example of generating HTDs now will be described. When an arbitraryimage is input, the input image is transformed to a frequency domain ofan orthogonal coordinate system or a polar coordinate system by Fouriertransformation. One-dimensional Fourier transformation following Radontransformation leads to transformation to the frequency domain of thepolar coordinate system. The Radon transformation refers to a series ofprocesses of obtaining 1-dimensional projection data by linearlyintegrating 2-dimensional images or multi-dimensional multimedia data atangles. In other words, the Radon transformation is based on theprinciple that an object appears differently depending on viewing anglesand the contour of the object can be estimated when viewed at everyangle.

The image transformed to the frequency domain of the polar coordinatesystem is filtered in a predetermined sub-frequency domain using aGarbor filter. The Garbor filter may consist of 5×6 filter regions, 5 ina radial direction and 6 at angles.

Next, in the frequency domain of the orthogonal coordinate system or thepolar coordinate system, texture features of the Garbor-filtered imageare extracted. Here, the frequency domain of the orthogonal coordinatesystem or the polar coordinate system can be divided into sub-frequencydomains, which are referred to as feature channels, based on the humanvisual perception system.

The texture features of the image include the mean and the variance ofenergies of the sub-frequency domains of the Garbor-filtered image, theenergy of a DC channel, and the variance of all pixel values of theimage.

A HTD can be expressed by formula (1) below using the mean and thevariance of energies for each frequency domain, i.e., each channel.F=[f _(STD) , f _(DC) , e(1), . . . , e(30), d(1), . . . , d(30)]  (1)wherein f_(STD) represents the variance of all pixel values of theimage, f_(DC) represents the energy of a DC channel, e(i), where i=1,CDOTS, 30, represents the mean of energies of each Garbor-filteredchannel, and d(i), where i=1, CDOTS, 30, represents the variance ofenergies of each Garbor-filtered channel.

In an embodiment according to the present invention, a HTD for eachframe or picture, which is expressed as formula (1) above, consists of62 values in Table 1 below. Homogeneous Texture HT[0]-Average HT[1]-STDHT[2]-Energy 1 HT[3]-Energy 2 HT[4]-Energy 3 . . . HT[60]-Energy STD 29HT[61]-Energy STD 30

Once the HTDs for all frames constituting the GoF or all picturesconstituting the GoP have been obtained, the GoF or GoP is expressedusing a predetermined representative HTD corresponding to one frame orpicture to reduce the amount of data (operation 150).

FIG. 2 is a schematic view for explaining a process of expressing aGoF/GoP using one representative HTD to reduce the amount of data.According to the present invention, this process can be achieved usingthree methods.

In a first method, the representative HTD includes the average of thevalues of each of the components of the HTDs for all frames or pictures.This can be expressed as formula (2) below. $\begin{matrix}\begin{matrix}{{{A\quad\upsilon\quad g\quad{{HT}_{k}(j)}} = {{1/M}\quad{\sum\limits_{i = b_{k}}^{e_{k}}{{HT}_{i}(j)}}}},} & {{j = 0},\cdots\quad,61}\end{matrix} & (2)\end{matrix}$wherein AvgHT_(k)(j) represents the average of the values of a j^(th)component in a k^(th) video shot, HT_(i)(j) represents the average ofthe values of the j^(th) component of homogeneous texture D in a i^(th)frame, b_(k) represents the first frame of the shot, e_(k) representsthe last frame of the shot; and M represents the number of frames in theshot.

In a second method, the representative HTD includes the median among thevalues of each of the components of the HTDs for all frames or pictures.This can be expressed as formula (3) below.MedianHT _(k)(j)=median{HT _(b) _(k+) ,(.), HT _(e) _(k−) ,(.), HT _(e)_(k) }, j=0, . . . , 61   (3)wherein MedianHT_(k)(j) represents the median among the values of thej^(th) component in a k^(th) video shot, HT_(i)(j) represents the j^(th)component of homogeneous texture D in the i^(th) frame, b_(k) representsthe first frame of the shot, and e_(k) represents the last frame of theshot.

In a third method, the representative HTD includes the smallest amongthe values of each of the components of the HTDs for all frames orpictures. This can be expressed by formula (4) below.IntHT _(k)(j)=min{HT _(i)(j)}, iε[b _(k) ,e _(k) ], j=0, . . . , 61  (4)wherein IntHT_(k)(j) represents the value of a j^(th) intersection in ak^(th) video shot, HT_(i)(j) represents the j^(th) component ofhomogeneous texture D in the i^(th) frame, b_(k) represents the firstframe of the shot, and e_(k) represents the last frame of the shot.

The syntax of DDL that can be used in the homogeneous-descriptor basedGoF/GoP texture description method according to the present invention isas follows. <!-- ################################################### --><!-- D efinition of MPEG-7 GofGopFeature --> <!--################################################### --> <complexTypename= GofGopFeature > <complexContent> <element name= D escriptor xsi:type= m peg7:VisualDType /> <attribute name= a ggregation u se= optional > <simpleType> <restriction base= s tring > <enumeration value=A verage /> <enumeration value= M edian /> <enumeration value= SplitMerge /> </restriction> </simpleType> </attribute> </complexContent></complexType>

The syntax of binary expression that is used in thehomogeneous-descriptor based GoF/GoP texture description methodaccording to the present invention is in Table 2 below. TABLE 2GofGopFeature{ Number of bits Mnemonic AggregationFlag 1 bsbf if(AggregationFlag){ AggregationType 3 bsbf } DescriptorID 8 uimsbfSizeOfDescriptor 8 uimsbf Descriptor bsbf }

Semantics of the above descriptor will be descried briefly.

The DescriptorID field defines a descriptor identifier using a binarynumber. Examples of descriptors include Color Layout, Dominant Color,Edge Histogram, Homogeneous Texture, etc. For example, the ID of ColorLayout can be 2, the Id of Dominant Color can be 7, the ID of EdgeHistogram can be 8, the ID of Homogeneous Texture can be 12.

The sizeOfDescriptor field defines the size of a descriptor using abinary number.

The Descriptor field represents an elementary feature using adescription tool defined in ISO/IEC 15938-3. Eight bits are assigned tothis filed as in the SizeOfDescriptor field.

The AggregationFlag field represents the presence of an aggregationattribute. Average, Median, SplitMerge, etc., belong to the aggregationattribute.

FIG. 3 is a block diagram of a GoF/GoP retrieval apparatus using thehomogeneous descriptor according to the present invention. The GoF/GoPretrieval apparatus includes a HTD database 300, a query HTD generationunit 320, an HTD search unit 340, and a GoF/GoP retrieving unit 360.

The HTD database 300 stores HTDs of a plurality of GoFs's/GoPs's,wherein the HTD of each GoF/GoP is expressed using a predeterminedrepresentative HTD corresponding to one frame/picture to reduce theamount of data. The query HTD generating unit 320 generates apredetermined HTD of a query GoF/GoP when the query GoF/GoP is input.

The processes of expressing the GoF/GoP using the predeterminedrepresentative HTD in the HTD database 300 and the query HTD generatingunit 320 are the same as described above in connection with the GoF/GoPtexture description method according to the present invention.

The HTD search unit 340 searches HTDs that are similar to the HTD of thequery GoF/GoP throughout the HTD database 300. The GoF/GoP retrievingunit 360 retrieves GoFs's/GoPs's corresponding to the searched similarHTDs and arranges the GoFs's/GoPs's in the order of degree ofsimilarity.

FIG. 4 is a flowchart of a HTD-based GoF/GoP retrieval method accordingto the present invention. The operation of the GoF/GoP retrievalapparatus using HTDs according to the present invention will bedescribed with reference to FIG. 3.

Initially, a database of HTDs for a plurality of GoFs's/GoPs's isestablished, wherein each GoF/GoP is expressed using one HTD (operation400). When a query GoF or GoP is input (operation 420), the query HTDgenerating unit 320 generates a HTD corresponding to one frame orpicture from HTDs of the query GoF/GoP (operation 440). The HTD searchunit 340 searches HTDs that are similar to the HTD of the query GoF/GoPin the HTD database 300 (operation 460). Finally, the GoF/GoP retrievingunit 360 retrieves GoFs's/GoPs's corresponding to the similar HTDs andarranges the GoFs's/GoPs's in the order of degree of similarity(operation 480).

The HTD-based GoF/GoP retrieval method and apparatus according to thepresent invention were experimentally tested using a dataset. Theresults are as follows.

The dataset used includes some of dataset and queries defined inM5124(ISISO/IEC JTC1/SC29/WG11 M5124 “Core Experiment onGroup-of-Frames/Pictures Histogram Descriptors(CT7)”) and new datasetand queries. They are composed of 1822 shots with 33 queries as definedin M9811 (ISO/IEC JTC1/SC29//WG11 M9811 “Dataset and Ground Truth setfor VCE-3 (Definition and use of a new “TimeSequence data Container”) ).

The video sequence used is as follows: misc1.mpg(CD20), misc2.mpg(CD21),camiloefilho.mpg(CD22), news2.mpg(CD18), basket.mpg(CD26),nhkvideo.mpg(CD26), Igerca_Lisa_(—)1.mpg(CD31), andIgerca_Lisa_(—)2.mpg(CD32).

In addition to the above dataset, a video sequence of culture.mpg(CD33),lascaux_english_sample.mpg (CD31), and tree1.mpg, which reflect thecharacteristic of texture distribution, was added.

The results of ANMRR using the HTD are shown in Table 3. TABLE 3 AverageMedian Intersection ANMRR 0.128 0.118 0.242

As is apparent from Table 3, the first method using average and thesecond method using median show higher search performances.

The texture of images can be more accurately expressed and can be moreefficiency and rapidly searched when using a GoF/GoP texture descriptionmethod and a GoF/GoP retrieval method and apparatus according to thepresent invention.

The invention may be embodied as computer readable codes in a computerreadable medium including all type of recording devices storing datareadable by any information processing device such as a computer.Examples of the computer readable medium include ROM's, RAM's, CD-ROMs,magnetic tapes, floppy disks, optical data storage devices.

While the present invention has been particularly shown and describedwith reference to exemplary embodiments thereof, it will be understoodby those of ordinary skill in the art that various changes in form anddetails may be made therein without departing from the spirit and scopeof the present invention as defined by the following claims.

1. A method of describing texture of a group of frames (GoF) or a groupof pictures (GoP), the method comprising: generating homogeneous texturedescriptors of all frames constituting the GoF or all picturesconstituting the GoF; and expressing the GoF or GoP using apredetermined representative homogeneous texture descriptorcorresponding to one frame or picture to reduce the amount of data. 2.The method of claim 1, wherein components of each of the homogeneoustexture descriptors include the mean and the variance of energies offrequency domains of a Garbor-filtered image, the energy of a DCchannel, and the variance of all pixel values of the image.
 3. Themethod of claim 2, wherein the homogeneous texture descriptors areexpressed as:F=[f _(STD) , f _(DC) , e(1), . . . , e(30), d(1), . . . , d(30)]whereinf_(STD) represents the variance of all pixel values of the image, f_(DC)represents the energy of the DC channel, e(i), where i=1, CDOTS, 30,represents the mean of energies of each Garbor-filtered channel, andd(i), where i=1, CDOTS, 30, represents the variance of energies of eachGarbor-filtered channel.
 4. The method of claim 1, wherein thepredetermined representative homogeneous texture descriptor in theexpressing the GoF or GoP includes the average of the values of each ofthe components of the homogeneous texture descriptors for all the framesor pictures.
 5. The method of claim 1, wherein the predeterminedrepresentative homogeneous texture descriptor in the expressing the GoFor GoP includes the median among the values of each of the components ofthe homogeneous texture descriptors for all the frames or pictures. 6.The method of claim 1, wherein the predetermined representativehomogeneous texture descriptor in the expressing the GoF or GoP includesthe smallest among the values of each of the components of thehomogeneous texture descriptors for all the frames or pictures.
 7. Amethod of retrieving a group of frames (GoF) or a group of pictures(GoP) that is similar to a query GoF or GoP, the method comprising:establishing a database of homogeneous texture descriptors of aplurality of GoFs's or GoPs's, each GoF or GoP being expressed by apredetermined representative homogeneous texture descriptorcorresponding to one frame or picture to reduce the amount of data;generating a homogeneous texture descriptor corresponding to one frameor picture of a query GoF or GoP when the query GoF or GoP is input;searching homogeneous texture descriptors that are similar to thehomogeneous texture descriptor of the query GoF or GoP in the database;and retrieving GoFs's or GoPs's corresponding to the searched similarhomogeneous texture descriptors and arranging GoFs's or GoPs's in theorder of degree of similarity.
 8. The method of claim 7, wherein thepredetermined representative homogeneous texture descriptor in theestablishing the database includes the average of the values of each ofthe components of the homogeneous texture descriptors for all the framesor pictures.
 9. The method of claim 7, wherein the predeterminedrepresentative homogeneous texture descriptor in the establishing thedatabase includes the median among the values of each of the componentsof the homogeneous texture descriptors for all the frames or pictures.10. The method of claim 7, wherein the predetermined representativehomogeneous texture descriptor in the establishing the database includesthe smallest among the values of each of the components of thehomogeneous texture descriptors for all the frames or pictures.
 11. Anapparatus for retrieving a group of frames (GoF) or a group of pictures(GoP) that is similar to a query GoF or GoP, the apparatus comprising: ahomogeneous texture descriptor database storing homogeneous texturedescriptors of a plurality of GoFs's or GoPs's, each GoF or GoP beingexpressed by a predetermined representative homogeneous texturedescriptor corresponding to one frame or picture to reduce the amount ofdata; a query homogeneous texture descriptor generating unit generatinga predetermined homogeneous texture descriptor of a query GoF or GoPwhen the query GoF or GoP is input; a homogeneous texture descriptorsearch unit searching the database for homogeneous texture descriptorsthat are similar to the predetermined homogeneous texture descriptor ofthe query GoF or GoP; and a GoF/GoF retrieving unit retrieving GoFs's orGoPs's corresponding to the searched similar homogeneous texturedescriptors and arranging the GoFs's or GoPs's in the order of degree ofsimilarity.
 12. A computer readable medium having embodied thereon acomputer program for the method of claim
 1. 13. A computer readablemedium having embodied thereon a computer program for the method ofclaim
 2. 14. A computer readable medium having embodied thereon acomputer program for the method of claim
 3. 15. A computer readablemedium having embodied thereon a computer program for the method ofclaim
 4. 16. A computer readable medium having embodied thereon acomputer program for the method of claim
 5. 17. A computer readablemedium having embodied thereon a computer program for the method ofclaim
 6. 18. A computer readable medium having embodied thereon acomputer program for the method of claim
 7. 19. A computer readablemedium having embodied thereon a computer program for the method ofclaim
 8. 20. A computer readable medium having embodied thereon acomputer program for the method of claim
 9. 21. A computer readablemedium having embodied thereon a computer program for the method ofclaim 10.